Extending Hidden Markov (tree) models for word representations

There is ample research in natural language processing (NLP) on obtaining word representations, including vector space modeling, clustering and techniques derived from language models. Good word representations are vital for overcoming the lexical sparseness inherent to many NLP problems. Much less studied are approaches capturing wider or global context (see e.g. Nepal and Yates (2014)). We are interested in using syntax for learning semantic classes, by which a wider and more relevant context can be incorporated (Šuster & Van Noord, 2014). It has been shown that dependency trees can extend a) Hidden Markov Models (HMM) in a way that resulting word representations increase performance in NLP classification tasks (Grave et al., 2013), but also b) Brown clusters, resulting in higher similarity scores in a wordnet-based experiment (Šuster & Van Noord, 2014). A drawback of the existing approaches is that trees are exploited only partially—dependency links set the structure (word context), but the identity of dependency links is not part of the model.